Generating Reproducible Statistical Analyses and Evaluation Reports: Principles, Practices, and Free Software Tools

Demonstration at Evaluation 2024: Amplifying and Empowering Voices, annual conference of the American Evaluation Association, Portland, OR

Steven J. Pierce

Center for Statistical Training and Consulting, Michigan State University

2024-10-22

Outline

What do you mean by reproducible?
Why should we aim for reproducibility?
How do we achieve reproducibility?

What do you mean by
reproducible?

Let’s define some concepts.

Reproducible Research (RR)

… is achieved when investigators share all the materials required to exactly recreate the findings so that others can verify them or conduct alternative analyses.

Statistical Results Can Be:

Repeatable

Original analyst
Original data

Reproducible

New analyst
Original data

Replicable

New analyst
New data

RR is a Product

RR is a product of how we work, not which methods we use.

Why aim for
reproducibility?

Several forces are promoting and enabling the push toward reproducibility.

Methods Matter!

Irreproducible < Reproducible < Replicated

Important

Reproducibility is an attainable minimum standard for science^[1].

Remember that our methods matter: how we conduct studies influences where our results fall on a continuum of trustworthiness and scientific credibility. Studies that use good scientific practices have higher trustworthiness and credibility. Let’s link this continuum to reproducibility.

At the low end, we have IRR, which relies on investigator reputation to establish credibility because key methodological details aren’t published and the findings can’t be independently verified. Reviewers and readers have limited ability to judge the quality of the research. IRR provides a poor foundation for future replication studies.

RR findings can be independently verified from the original data, reducing reliance on investigator reputation. Reviewers and readers can better judge quality of research on its own merits. Furthermore, RR provides a solid foundation for future replication studies by publishing key methodological details.

Finally, replicated research has the highest level of credibility because independent investigators have replicated the findings with new studies, indicating that the probability of opportunistic biases is low and the findings generalize beyond the original sample. This dramatically increases trustworthiness of a finding.

Roger Peng has argued that RR is an attainable minimum standard for science (Peng et al., 2006).

Guiding Principles for Evaluators

Pursuing reproducibility enacts our guiding principles^[2]:

AEA logo.

Systematic inquiry
Competence
Integrity

Funders Value RR

Data sharing and reproducibility initiatives

NIH logo. NSF logo.

IES logo.

Emerging Scientific Norms

Open science:
- https://www.cos.io/open-science
- https://open.science.gov
Pre-registered methods^[3–5]
Reporting guidelines^[6–9]
Publishing raw data^[10,11]
Journal & publisher policies^[4,10–13]
Institutional policies

Publishing Technology

Career Benefits of RR

Motivation to focus on quality
Become more efficient
Create more products
Easier to get published
Get cited more often
Build your reputation

How do we achieve reproducibility?

Understand the criteria, then apply principles, practices, & tools.

Criteria for Reproducbility
^[1]

Data
Methods
Documentation
Distribution

Materials Required to Recreate Findings

Materials	Findings
Manuals & procedures	Statistics
Instruments & scoring rules	Coefficients & p-values
Codebooks	Confidence intervals
Methods applied	Effect sizes
Data mgt decisions	Model fit indices
Data files	Figures
Software & analysis scripts	Tables

Principles for Achieving Reproducibility

Collaboration
Organization
Automation

Preservation
Integration
Separation

Understanding a few guiding principles helps people figure out how to do RR. Here are the principles I’ve identified so far. Thinking about reproducibility as collaboration encourages us to prepare and share materials that will make our work accessible to others. Organization simplifies communicating to others how to reproduce our work. Automation helps us achieve efficiency and accuracy because scripts can be faster and more reliable than manual processes. The preservation principle guides us to keep our raw data pristine and to keep a good audit trail. The integration principle suggests building documentation directly into data files, putting comments in code, and using code to directly insert statistical results into reports. The separation principle helps us manage complexity by splitting projects or tasks into pieces; for example separating data management from analysis, or keeping raw data separate from working data files used for specific analyses. Keep an eye out for instances of these principles in action throughout the rest of this talk. My examples will aim toward the middle of the complexity spectrum.

Workflow Woes

Use GUI, menus, & dialog boxes to do tasks
Manually updating data files
Version control through saving to new file names
Disorganized folders & files
No audit trail
Copy & paste output from stats software to word processor
Fixing mistakes took lots of time

As a student, I was taught a manual workflow for doing data analysis and writing reports. We used the GUI, menus, and dialog boxes in SPSS to do data management and analysis. We updated data files manually. We created tons of files, saving to a new filename as changes accrued. That led to confusing proliferation of folders and files with little planned structure. It was hard to tell who had last edited various files and which versions to use. We retyped or pasted results from SPSS output into Word documents then spent lots of time formatting them. If we made a mistake, it could takes hours or even days to re-do things. Over time I learned some strategies to improve that workflow, such as using SPSS syntax files to automate tasks, adding comments in the syntax files, adopting file naming conventions, and so on. But part of the problem was that the software tools weren’t integrated and lacked support for better workflows.

Spock

“Captain, you’re asking me to work with equipment which is hardly very far ahead of stone knives and bearskins.”

Star Trek (1966) - S01E28 The City on the Edge of Forever

Free, Open Source Software Tools for Reproducible Research

R^[14], for statistical computing
RStudio Desktop^[15], an editor
Quarto^[16], for dynamic documents
Git^[17], for version control
GitHub.com, for sharing & collaborating on code

Dynamic Documents Via Quarto + R

Dynamic documents are a crucial technology for reproducibility. They implement the concept of literate programming, where one embeds code directly in a document such as a report, interleaving it with narrative text, headings, and in-text citations. When you render dynamic documents, the code is executed and the results are directly inserted into the output. That includes statistics computed from the data, figures, tables, formatted equations, fully-formatted reference sections, and more. This diagram shows the files involved for a simple example. The idea is that rendering your report script produces an output such as a PDF file ready for delivery to your client.

Several principles are in action here: Separation of data from code, integration of R code into the Quarto script, where it automates data management and analyses. Quarto automates inserting results into the report. The BibTeX file contains data about references you cite in the report, while the CSL file tells QAuarto how to format the citations and the reference list. I’m using similar file names to stay organized: they show which script creates the report. Saving all the files follows the preservation principle. One can rapidly reproduce the report by re-rendering the script, or fix an error in the script then render again to automate updating the report.

Research Compendium

A compendium organizes digital files so others can review or use them to reproduce results, or do new analyses.^[18] It should:

Use prevailing conventions to organize the project
Separate data, method, & output but show how they are related
Specify the software used in the original analysis

R Packages

… are folders of digital files designed for sharing code & help documentation.^[19] They

Organize files into a conventional structure
Can contain data, meta-data, & other documentation
Can contain dynamic documents & rendered output

Tip

Use an R package for a research compendium!^[18]

Git Repositories

… are folders of files being tracked by Git^[17] for version control purposes.^[20,21] They:

Preserve a history of changes to each tracked file
Allow recovering a file’s prior state from the history
Can be local or remote (hosted on a server)
Can be private or public

Tip

Put your research compendium in a Git repository!

Git is software for doing version control on code and other files stored in repositories, which are just–you guessed it–digital folders. You can choose which files to track, then log of all committed changes to them, when they were made, who made them, and why. That preserves a fantastic audit trail. You can recover the exact state of a file as it existed at any commit in the history. You can make repositories either private or public. I usually keep a repository private until the corresponding paper has been published, then I can make it public to enhance reproducibility. So, putting your compendium in a Git repository both follows some of the principles I’ve highlighted and helps solve some of those worklflow woes I described earlier. For example, you can stop using long, confusing filenames as a crude form of version control. Give each file a simple name that stays constant even while Git tracks how it changes over time. Jenny Bryan’s paper on using proper version control persuaded me to start using Git. I have not regretted it! So, put your compendium into a Git repository!

GitHub

While Git handles preservation, GitHub solves other workflow problems. GitHub is an online server that hosts Git repositories, allowing multiple people to clone copies from the remote server to their own computers and synchronize their local copies with the main one hosted on GitHub. This facilitates collaboration because each person can independently edit files in their own local copy, pull the recent changes others have made down from GitHub, and push their own changes back up to the main repository. Git will detect conflicting changes and force a user to resolve them and commit the result.

Using a private GitHub repository while working on the project allows you collaborate with teammates as you develop a report. Later, you can make the repository public so that people outside your team can access reproducibility materials.

Collaboration

Plan to give files to multiple stakeholders:

Your future self
Your team
Peer reviewers
Your audience

Organization

Organize files into a research compendium.
Create documentation (README files, vignettes, etc.).
Use folder structures, naming conventions,
Use data flow diagrams and/or rendering scripts

Automation

Use scripts instead of GUIs
Make computers do the tedious labor
Computers are patient idiots

Preservation

What should you preserve?

Data and meta-data
Methodology decisions
Data cleaning, management, and analysis scripts
Key decisions and rationales
Source code for scripts
Version histories of data, code, output, and deliverables
Who changed what & when
Software environment & versions used

Integration

Attach meta-data directly to data
Include codebooks (or scripts to generate them from the data)
Integrate narrative text with code via Markup languages
Add custom functions and associated help files to research compendium
Include references

Separation

Data from code
Raw data from cleaned data
Code from output
Draft from production output
Version history from current files

Version Control

Git tracks how files changed (additions & deletions), who changed them, when, & why
Searchable & recoverable version history
Eliminates the need to use filename conventions for versioning

Narrative Text

Formatting
Headings
Links

Inline R Code

Tables

Figures

Parameterized Reports

Map Your Path to Reproducibility

References

1. Peng, R. D., Dominici, F., & Zeger, S. L. (2006). Reproducible epidemiologic research. American Journal of Epidemiology, 163(9), 783–789. https://doi.org/10.1093/aje/kwj093

2. American Evaluation Association. (2018). Guiding principles for evaluators [Web Page]. Author. https://www.eval.org/About/Guiding-Principles

3. Bosnjak, M., Fiebach, C. J., Mellor, D., Mueller, S., O’Connor, D. B., Oswald, F. L., & Sokol, R. I. (2022). A template for preregistration of quantitative research in psychology: Report of the Joint Psychological Societies Preregistration Task Force. American Psychologist, 77(4), 602–615. https://doi.org/10.1037/amp0000879

4. DeCoster, J., Sparks, E. A., Sparks, J. C., Sparks, G. G., & Sparks, C. W. (2015). Opportunistic biases: Their origins, effects, and an integrated solution. American Psychologist, 70(6), 499–514. https://doi.org/10.1037/a0039191

5. Moore, D. A. (2016). Preregister if you want to. American Psychologist, 71(3), 238–239. https://doi.org/10.1037/a0040195

6. Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., & Rao, S. M. (2018). Journal article reporting standards for quantitative research in psychology: The APA publications and communications board task force report. American Psychologist, 73(1), 3–25. https://doi.org/10.1037/amp0000191

7. Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gøtzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010). CONSORT 2010 explanation and elaboration: Updated guidelines for reporting parallel group randomized trials. Journal of Clinical Epidemiology, 63(8), e1–e37. https://doi.org/10.1016/j.jclinepi.2010.03.004

8. Schulz, K. F., Altman, D. G., & Moher, D. (2010). CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials. Journal of Clinical Epidemiology, 63(8), 834–840. https://doi.org/10.1016/j.jclinepi.2010.02.005

9. von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007). The strengthening the reporting of observational studies in epidemiology (STROBE) statement: Guidelines for reporting observational studies. PLoS Medicine, 4(10), e296. https://doi.org/10.1371/journal.pmed.0040296

10. Hrynaszkiewicz, I., & Altman, D. G. (2009). Towards agreement on best practice for publishing raw clinical trial data. Trials, 10(1), 17. https://doi.org/10.1186/1745-6215-10-17

11. Hrynaszkiewicz, I., Norton, M. L., Vickers, A. J., & Altman, D. G. (2010). Preparing raw clinical data for publication: Guidance for journal editors, authors, and peer reviewers. Trials, 11(1), 9. https://doi.org/10.1186/1745-6215-11-9

12. Laine, C., Goodman, S. N., Griswold, M. E., & Sox, H. C. (2007). Reproducible research: Moving toward research the public can really trust. Annals of Internal Medicine, 146(6), 450–453. https://doi.org/10.7326/0003-4819-146-6-200703200-00154

13. Peng, R. D. (2009). Reproducible research and biostatistics. Biostatistics, 10(3), 405–408. https://doi.org/10.1093/biostatistics/kxp014

14. R Development Core Team. (2024). R: A language and environment for statistical computing (Version 4.4.1) [Computer Program]. R Foundation for Statistical Computing. http://www.R-project.org.

15. RStudio Team. (2024). RStudio Desktop: Integrated development environment for R (Version 2024.04.2+764) [Computer Program]. Posit Software, PBC. https://posit.co

16. Allaire, J. J., Dervieux, C., Scheidegger, C., Teague, C., & Xie, Y. (2024). Quarto (Version 1.5.57) [Computer Program]. Posit Software, PBC. https://quarto.org

17. Torvalds, L., Hamano, J. C., & other contributors to the Git Project. (2024). Git for Windows (Version 2.46.0(1)) [Computer Program]. Software Freedom Conservancy. https://git-scm.com

18. Marwick, B., Boettiger, C., & Mullen, L. (2018). Packaging data analytical work reproducibly using r (and friends). The American Statistician, 72(1), 80–88. https://doi.org/10.1080/00031305.2017.1375986

19. Wickham, H., & Bryan, J. (2021). R packages: Organize, test, document, and share your code. O’Reilly Media. https://r-pkgs.org

20. Bryan, J. (2018). Excuse me, do you have a moment to talk about version control? The American Statistician, 72(1), 20–27. https://doi.org/10.1080/00031305.2017.1399928

21. Chacon, S., & Straub, B. (2014). Pro Git. Apress Media. https://git-scm.com/book/en/v2